Validity of the three parameter item response theory model for eld test data ITP Research Series

Item response theory is a large sample procedure to estimate item parameters based on individual response strings. However, what happens when the data available to estimate item parameters is small? This situation is common when new assessment items are tried out for inclusion in future operational assessments, commonly called field testing. In field tests, many items are spread out over a fixed set of respondents which can limit the number of responses on a given field test item. Four models are compared with real world field test data to evaluate their ability to accurately estimate item parameters in order to inform test developers. Implications for the four models on estimating field test item parameters are discussed. Birnbaum’s three parameter logistic item response theory (3PL IRT) model is a widely used model for assessment data (Birnbaum, 1968). The wide use of this model stems from the flexibility this model offers. The 3PL IRT explicitly allows for the test questions (commonly called items) to have varying discrimination parameter estimates across the items and also account for the nonzero likelihood of answering the item correctly by guessing (De Ayala, 2013). In addition, the model flexibility can be shown by the better fit compared to simpler models such as the two parameter (2PL) or Rasch model (CTB/McGraw Hill, 2008) that fix some of the IRT parameters to specific values (i.e. zero of one) instead of estimating them. However, a limitation of the 3PL IRT model is the larger sample sizes required to estimate the three parameters for every item (De Ayala, 2013). This limitation can increase the uncertainty in the parameter estimates, particularly for the pseudo-guessing parameter, in small sample size conditions (Thissen & Wainer, 1982). For operational forms, sample size is commonly not a concern. However, when trying out new items for inclusion in a future operational forms, commonly called field testing or tryouts, the sample size can become much smaller where uncertainty in parameter estimates may be a problem.